Caches were invented for computer systems to address the problems associated with slow memory access relative to processor speed. Without a cache, the processor must wait for instructions or operands (“data”) from memory in order to proceed with program execution. The time a processor spends waiting essentially wastes processor capacity. With a cache, the long memory access times are eliminated when the requested data are present in the cache.
While memory speeds have improved substantially, the increase in processor speeds has been far greater. The increase in memory speed means that a processor will wait less (real time) for data to be retrieved from memory. However, the increased processor speed means the processor must wait a greater number of processor cycles in reading the data from memory. Thus, even though a processor may spend less real time waiting for data, the processor may be spending a greater number of cycles waiting relative to prior generations of processors. While processor speeds have improved, the processors may be spending a greater percentage of time waiting and therefore wasting a greater percentage of processing capacity.
While increased cache sizes reduce the number of times that a process is forced to wait for data from memory, the increased size alone is not enough to make up for the growing difference in speeds between memories and processors.
A method and apparatus that address the aforementioned problems, as well as other related problems, are therefore desirable.